Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix setindex! with SubDArray source (+ remove Julia 0.4 support) #76

Merged
merged 8 commits into from
Aug 4, 2016

Conversation

andreasnoack
Copy link
Member

Supersedes #74

This method is an optimization wherein we try to chunk accesses based upon the parent DArray's parts. The hard thing is then going backwards and trying to figure out which parts of the assignment indices need to be used in order to access those chunks.  This is a four stage process that uses five different types of indices:

1. Find the indices of each portion of the DArray
2. Find the valid subset of indices of the SubArray that index into that portion
3. Find the portion of the indices for the assignment that need to be used for that subset of indices in step 2. This is the hard part.  It requires creating another set of indices that represents the mask of valid indices from step 2.  With those masks in hand, it's possible to reindex `I` to the indices we need. The trouble is that `setindex!` supports singleton dimensions in the source array in ways that `getindex` does not, so we need to selectively drop singleton dimensions as we restrict the indices. A final complication is that the last index can be a linear index over many indices in either the source or destination.
4. Finally, if the entire DArray chunk isn't getting used, we need to shift the indices from step 2 to refer to the local part of the DArray.
This is no longer needed -- the comment is from when I only had restrict_indices partially implemented
Also clarify the comment since I was confused upon coming back to this method a few weeks later
Both these lazy arrays are effectively generalizations of Tim's MappedArrays.jl package. Doing this generally adds a bit more difficulty in terms of element types, but that is true of the MappedArray type, too.  It might be worth breaking this out into a package at some point.
As a further optimization, (at)inbounds could be added throughout the algorithm once it has received more widespread testing.
@codecov-io
Copy link

codecov-io commented Aug 3, 2016

Current coverage is 67.93% (diff: 78.00%)

Merging #76 into master will decrease coverage by 0.44%

@@             master        #76   diff @@
==========================================
  Files             1          1          
  Lines           702        711     +9   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits            480        483     +3   
- Misses          222        228     +6   
  Partials          0          0          

Powered by Codecov. Last update 166642e...feff2c6

Remove 0.4 support
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants